The present invention pertains to graphical user interfaces and, more particularly, to creating a user-customizable system and method to access, navigate and control the screens, objects and applications on computer devices (desktops, laptops, smart phones, tablets.) by inventing a new system and method that empowers computer users to enjoy easy and productive mouse-free and touch-free control of and access to their computers and all of their applications.
Computers commonly use a Graphical User Interface (GUI) to display various graphical interface objects on a display device. Examples of GUI objects are windows, menus, icons, text entry fields, buttons, check-boxes, drop-down lists and HTML links. Many of these objects can then be interacted with, acting as an interface for the computer and/or its currently running application.
As an example, consider an email client. The email client is made up of different objects contained in a Window. Within the email client the Window itself is a top level GUI object and can contains sub-objects such as a textbox object where a user can type an email message and a button object that the user can click on to send an email. Typically the user would identify the send button by a label object overlaid on top of the button objects which may read “send” or synonymous terms.
These GUI objects are typically arranged in a hierarchy resembling a tree like structure. The screen real estate of a monitor is typically abstracted as a desktop in which running applications can display any number of windows. Each window also typically consists of a hierarchy of controls such panes, buttons, labels, list items. GUI objects such as a button can also be hierarchical and can contain an image and a label.
Applications designed for a GUI use their windows and their associated hierarchy of GUI objects to facilitate user interaction. As the user moves a mouse or finger across a computer desktop (virtualized area representing the total available screen real estate), the operating system typically broadcasts this information in a way in which applications can take action based on the location of an input device. For example, if a user clicks on a button object in a window, the operating system will broadcast that a click took place at a specific location and the application in which the button was drawn under the cursor will be able to use that broadcasted information along with its internally represented hierarchy of objects to determine that a mouse click did occur over a button object and then act upon that information by performing an action. With this example it's important to note that an application can act upon a button being clicked even though the application responsible for drawing a mouse cursor and tracking its location, and even the operating system itself, have no idea there is a button at that location.
The above represents one way an application can link its underlying functions to its GUI and while the specifics differ from device to device and operating system to operating system, the underlying paradigm remains the same: The operating system, or high level application running on a computer device (such as the Windows Desktop Manager) does not need to be aware of what is displayed by an application or how it facilitates interaction with its display but is merely responsible for creating a set of rules by which the application can request the information it needs to accomplish those goals.
This describes how an immense number of mouse (or touch) intensive applications operate. Users can only control these applications by pointing and clicking the mouse, or by touching and tapping. Such applications range from home and auto loan applications, web pages, intranet sites, credit card forms and government eligibility forms to medical billing and Electronic Medical Records (EMRs). Such applications are heavily used by the more than twenty million clerical workers (Office and Administrative Support) in the US, “Office and Administrative Support Occupations”, http://www.b1s.gov/oes/current/oes430000.htm, May 2013. In addition, other professional workers frequently access databases, document management systems and websites in their work. For example, doctors are increasingly dependent on EMRs
Many, if not most of these programs, are not accessible to users of speech recognition software and other accessibility software. An accessible program needs to fully communicate its internal representations of objects and details about how it can be interacted upon with other applications. This is typically done through an Accessibility Framework, which will be discuss in more detail shortly.
First, let's abstract this situation by considering all graphical objects in a GUI to belong to one or more of three hierarchies regardless of the device or operating system being used. There is the hierarchy maintained by the operating system and or accessibility frameworks (Hierarchy 1), the hierarchy maintained by the individual application (Hierarchy 2), and the visual hierarchy of controls observed visually by a sighted user with good vision (Hierarchy 3). In the above example of an email client the application's window was a member of all three hierarchies, but the send button was only a member of the second two. In this case the application is not accessible because the first hierarchy is not aware of the button and therefore other applications, without using the present invention, have no way of being aware of that object.
Ideally an application communicates its Hierarchy (Hierarchy 2) with the operating system's Hierarchy (Hierarchy 1) through the use of an accessibility framework such as the Microsoft UI Automation and Active Accessibility in the Window's operating system. Microsoft's Automation API is the Microsoft toolset for accessibility. Microsoft's UI Automation and Active Accessibility ensure that applications are accessible to people who use assistive technology products. When used properly, an application's inter-actable objects in Hierarchy 2 are presented to accessibility applications as exposed controls.
Unfortunately, accessibility frameworks such as the Microsoft UI Automation and Active Accessibility used by the Windows Operating System, are frequently ignored and applications fall into a range of inaccessibility which range from totally inaccessible to nearly accessible where nearly accessible means there are only minor differences between Hierarchies 1 and 3.
In order to present an object on the Desktop an application only needs to register a window and specify the window's size and other parameters with the operating system. How the window itself is composed by the application is up to the application. It is entirely possible, even common, that the application composes the window as it would a canvas: it just draws its interface (buttons, menus, labels etc.) into the window. In this case the user sees a typical window hierarchy (Hierarchy 3), but the operating system or the Desktop Manager is only aware of the window itself. Thus, Hierarchy 1 only consists of one item: the window, since the Window object is the only exposed control and all the application controls are either not exposed or inaccurately exposed.
While this represents a seemingly extreme scenario in which the operating system has no hierarchical representation of a window in an application, many applications fall into this completely inaccessible category. But, the opposite is rarely true, almost no applications fall into the completely accessible category with all three of the object hierarchies perfectly in sync.
When a computer operator uses a mouse to interact with an application that does not use an accessibility frame such Microsoft UI Automation and Active Accessibility guidelines or the equivalent on a different operating system/device, any discrepancies between Hierarchies 1, 2 and 3 aren't an issue to the user because the application itself can maintain a typical interaction by only receiving mouse events within the window and then internally deciding how to handle the object that was drawn on the canvas at the time of the event and perform a function consistent with the specified event.
But, when a computer user with disabilities tries to interact with an application that does not use an accessibility framework, the discrepancy between hierarchy 1 and 3 makes the software inaccessible to speech recognition applications and other assistive technology programs.
The lack of accessibility is a serious problem for people with disabilities. For example, users with hand or wrist injuries are unable to position a mouse and click a mouse button due to severe pain. In addition, mice and touch screens are not usable by disabled persons such as quadriplegics. Further, there is evidence that extensive use of mouse-intensive applications over an extended period of time could lead to physical injury to some users meaning they one day may fall into the category of those for whom mouse use is painful.
The software is unusable by a user with disabilities because the operating system is unable to inform accessibility software anything about what controls can be interacted with. Thus, buttons in an inaccessible application cannot be read by a screen reader, clicked by a speech recognition program, or its text properly magnified by a screen magnification tool.
There are tens of thousands of inaccessible or partially inaccessible applications including graphics and legacy applications used on the job by hundreds of thousands of government and corporate employees. Referring to the drawings, FIG. 1, depicts a partially inaccessible application. The application depicted exposes its system menu controls 101. 102, 103 to the operating system, Hierarchy 1. But, the application does not adhere to any accessibility guidelines to that the menus 104 and the controls 105, 106 in the program remain inaccessible to a user of assistive technologies. Computer users with disabilities are frequently confronted with inaccessible or partially inaccessible applications, templates, forms and databases as in FIG. 1. Inaccessible applications place persons with disabilities at a huge disadvantage in the workplace, presenting a needless barrier to their gainful employment, perhaps one reason why the unemployment rate among those with disabilities is twice that of those without disabilities.
These problems are not confined to the workplace. Even most instructional software is inaccessible. In a survey of twenty-five award winning companies who produce pre-college instructional software, not a single company was addressing accessibility in their product development “Designing-Software-Accessible-Individuals-Disabilites”, http://www.washington.edu/doit/Brochures/Technology/design_software.html, 2014. Thus, those with disabilities face even face barriers trying access and control software that can help them achieve a quality education.
The problem is getting worse. The barriers to entry in creating applications are being eradicated. It is now easy for non-technical people to create applications. But, only experienced programmers can make the application accessible. A typical example of this is the Microsoft Developer library, “Windows Automation API (Windows)”, https://msdn.microsoft.com/en-us/library/windows/desktop/ff486375(v=vs.85).aspx, 2015, states, “The Windows Automation API is designed for experienced C/C++ developers.”
The severity of the problem is evidenced by laws such as Section 508 in the ADA act, which has led to a set of guidelines such as “Software Applications and Operating Systems (1194.21)”, http://www.access-board.gov/guidelines-and-standards/communications-and-it/about-the-section-508-standards/guide-to-the-section-508-standards/software-applications-and-operating-systems-1194-21, Jun. 21, 2001. But, laws and guidelines haven't solved the problem, or even lessened the issue.
Since almost anyone can create an application and only experienced programmers can make the applications accessible, there has been an onslaught of inaccessible programs. The need for the present invention becomes clear. The barriers to making applications accessible need to be eradicated. Non-technical people need to be empowered to make applications accessible by letting them quickly bridge the gaps between hierarchies 1 and 3. The present invention provides a novel and comprehensive system and method to do this.
For relief, many users try to address their need for a mouse-free and touch-free graphical user interface by turning to assistive technologies such as speech recognition engines that ship with operating systems such as IOS, Android, Chrome, OSX, Windows, and third-party speech engines such as Dragon Naturally Speaking.
But, prior art technologies do not offer a simple, easy to use or efficient solution. For example, the best of these technologies, Dragon NaturallySpeaking, includes a widely used mouse grid. There are other mouse grid solutions like VoiceComputer's voice mouse. These mouse grid technologies let a user access a particular screen location on a computer device but these technologies were not designed for and are not used in order to drill down or navigate an application because these technologies are: 1) Too slow. Even a proficient user usually needs to issue four or more speech commands to click on a single control and, 2) Too awkward and confusing. Each of the 4+ speech commands needed to click on a specified point requires a mental calculation, “Dragon Naturally Speaking—Test 1”, http://athenpro.org/googledocsreports/dragon1.php, 2015, “Dragon Naturally Speaking—Test 2”, http://athenpro.org/googledocsreports/dragon2.php, 2015, “WebAIM Assistive Technology Experiment_Dragon NaturallySpeaking”, http://webaim.org/blog/at-experiment-dragon/, 2011.
Prior art inventions, for example U.S. Pat. No. 7,742,923 and U.S. Pat. No. 6,615,176, may improve a user's access to controls exposed by the active application in comparison to a mouse grid. That is, they improve access to Hierarchy I controls, controls that conform to an accessibility framework. But, the prior art inventions are not a solution and haven't been successful since they don't address the inaccessibility of the many programs that do not conform to an accessibility framework.
The failure of the prior art technology is demonstrated by the existence of an entire consulting business whereby experts create scripts (small programs usually written in a scripting language that are part of speech recognition program such as Dragon NaturallySpeaking and in screen reader programs such as JAWS) so that users can access various screen locations and overcome the limitations of the prior art. But, even this doesn't solve the problem. Creating these commands is time consuming and expensive, an expense that frequently runs into the thousands of dollars, and, significant difficulties remain even if one invests the time it takes to create a full set of commands for an application or web page. For one, applications are frequently updated, making much or all of previous customization obsolete. For two, many applications require hundreds, even thousands of uniquely named commands. Users cannot be expected to remember the names of hundreds of uniquely named commands, even after repeated use, thus, complicating and slowing down control.
In many cases, software applications cannot be made accessible with prior art technologies. Some examples, JMP, CWS/CMS, CALWIN and many applications that run on the Citrix desktop. These applications are so complex that it would take thousands of individual macros to control them. Such commands sets have never been made because they would be too complex to use.
In some applications like COMPASS, the menus and other controls are Hierarchy 3 controls drawn on the screen, so their controls are unexposed to Hierarchy 1 and the position of the controls change as the application is used making them inaccessible to prior art technology which uses mouse location data to determine where a control is located.
Most applications and websites are not designed for use by speech recognition, screen readers and other accessibility tools and do not fully confirm to any accessibility guidelines. Many applications, websites and web apps do not confirm to any accessibility guideline. Thus, users of accessibility software who have disabilities cannot fully access and control most of their applications.
What is needed is a system and method that delivers what users require to get their work done. Users require a computer application that would allow a computer user, including the 50-60 million people with disabilities in the United States, “Understanding Software Accessibility”, http://www.uspto.gov/about/offices/cio/section508/sw_accessibility.ppt, 2004 to easily and productively access, control and navigate all of their computer applications, not just applications that adhere to accessibility guidelines, with accessible technology such as Dragon NaturallySpeaking and JAWS.